Field of the Invention
The present invention relates to an information processing apparatus, an information processing method, and a system, particularly to a technique for presenting mixed reality.
Description of the Related Art
In the industry of manufacturing automobiles, aircrafts, and the like, there is a demand for reducing a period of designing processes and cost saving. One technique that satisfies such a demand is a mixed reality (MR) system that seamlessly composites a physical space with a virtual space. The mixed reality system can display a product under development as a computer graphics (CG) image superimposed over a physical world. In this way, the product is checked and examined from free viewpoints, which allows for examination of designs and the like without creating a full-scale model.
In the mixed reality system, a head-mounted display (HMD) including an image sensing device, such as a video camera, and a display in an integrated manner is used as an image display device. The HMD comes in various forms, one example of which is a video see-through method. This method displays an image of a virtual space generated in accordance with the position and orientation of the image sensing device mounted in the HMD, in such a manner that the image of the virtual space is superimposed over an image of a physical space captured by the image sensing device. Examples of the image of the virtual space include a virtual object rendered using CG, and character information. There is also an image display device using an optical see-through method. This method displays an image of a virtual space generated in accordance with the position and orientation of a viewpoint of an observer on an optically transmissive display that is worn on the head of a subject.
In the manufacturing industry, a review conducted by many people often takes place during designing processes. In a review that uses a mixed reality system, the system is configured by combining an HMD and hand-held information terminals, such as tablets, to allow many people to experience a mixed reality space (a space created by compositing a virtual space and a physical space). In this system, an image of the mixed reality space viewed by a person wearing the HMD is simultaneously distributed and displayed on a plurality of tablet screens. This allows a plurality of people to simultaneously share the view seen by the person wearing the HMD and participate in designing and examination.
Japanese Patent Laid-Open No. 2006-293604 discloses an example of cooperative work using a mixed reality system. In this example, a plurality of remote participants share a mixed reality space experienced by a worker, and cooperative work is enabled as they seamlessly perceive physical objects and virtual objects while changing a viewpoint(s).
On the other hand, Makita et al., “Mixed Reality Navigation on a Tablet Computer for Supporting Machine Maintenance in Wide-area Indoor Environment,” Proc. of ICServ2014, pp. 41-47, 2014 discloses a technique to switch between display in an augmented reality (AR) display mode and a virtual reality (VR) display mode in a navigation system. This technique can support actions of a user by enabling the user to appropriately grasp the surrounding situation. Furthermore, a commercial graphics library has been provided with a function of obtaining correspondence between arbitrary points on a screen (a plane of a displayed image) and components constituting a three-dimensional model in a virtual space through the reverse perspective projection conversion.
Assume a case in which a tablet user designates a component of a virtual object displayed on a screen of a tablet while the tablet is sharing an image with an HMD user in a mixed reality system. In this case, as the HMD user changes the position and orientation of a viewpoint by their own will, the tablet user who is sharing the image cannot control the position and orientation of the viewpoint with respect to the virtual object. That is to say, a display screen of the tablet changes independently of the intention of the tablet user, which makes it difficult to accurately designate a desired component.